Running Selenium functional test in TFS 2015 vNext repeats 'InProgress' and eventually times out

Question

I'm trying to run a Selenium test using the Run Functional Test step from a vNext build definition on TFS 2015, the test project's sources are built on the TFS server and copied to a remote machine running Windows 8.1. However, when it's to run the unit test, nothing happens with a repeated message " test is in state 'InProgress' " in the TFS console. On the DTAExecutionHost log file in the remote machine, the following lines are repeated (snippet from end of log file).

Web method running: [https://.com/tfs/DefaultCollection/_apis/test/Agents/19/Commands/0] (GET)0[test] Web method response: [https://.com/tfs/DefaultCollection/_apis/test/Agents/19/Commands/0] (GET)0[test] 15 ms V, 3872, 21, 2016/09/22, 13:45:05.769, KLPC-607V\DTAExecutionHost, TestExecutionServiceCommandQueueListener:DispatchAsync W, 3872, 21, 2016/09/22, 13:45:05.769, KLPC-607V\DTAExecutionHost, TestExecutionServiceCommandQueueListener.DispatchAsync : TestExecutionCommand received with None. Retrying it. W, 3872, 21, 2016/09/22, 13:45:05.769, KLPC-607V\DTAExecutionHost, TestExecutionServiceCommandQueueListener.DispatchAsync : Queue not found. Delaying for 50000 milliseconds. V, 3872, 18, 2016/09/22, 13:45:55.773, KLPC-607V\DTAExecutionHost, TestExecutionServiceCommandQueueListener.DispatchAsync : Getting Command for Test Agent Microsoft.TeamFoundation.Test.WebApi.TestAgent Web method running: [https://.com/tfs/DefaultCollection/_apis/test/Agents/19/Commands/0] (GET)0[test] Web method response: [https://.com/tfs/DefaultCollection/_apis/test/Agents/19/Commands/0] (GET)0[test] 15 ms V, 3872, 19, 2016/09/22, 13:45:55.788, KLPC-607V\DTAExecutionHost, TestExecutionServiceCommandQueueListener:DispatchAsync W, 3872, 19, 2016/09/22, 13:45:55.788, KLPC-607V\DTAExecutionHost, TestExecutionServiceCommandQueueListener.DispatchAsync : TestExecutionCommand received with None. Retrying it. W, 3872, 19, 2016/09/22, 13:45:55.788, KLPC-607V\DTAExecutionHost, TestExecutionServiceCommandQueueListener.DispatchAsync : Queue not found. Delaying for 50000 milliseconds. V, 3872, 17, 2016/09/22, 13:46:45.792, KLPC-607V\DTAExecutionHost, TestExecutionServiceCommandQueueListener.DispatchAsync : Getting Command for Test Agent Microsoft.TeamFoundation.Test.WebApi.TestAgent Web method running: [https://.com/tfs/DefaultCollection/_apis/test/Agents/19/Commands/0] (GET)0[test] Web method response: [https://.com/tfs/DefaultCollection/_apis/test/Agents/19/Commands/0] (GET)0[test] 16 ms V, 3872, 16, 2016/09/22, 13:46:45.808, KLPC-607V\DTAExecutionHost, TestExecutionServiceCommandQueueListener:DispatchAsync W, 3872, 16, 2016/09/22, 13:46:45.808, KLPC-607V\DTAExecutionHost, TestExecutionServiceCommandQueueListener.DispatchAsync : TestExecutionCommand received with None. Retrying it. W, 3872, 16, 2016/09/22, 13:46:45.808, KLPC-607V\DTAExecutionHost, TestExecutionServiceCommandQueueListener.DispatchAsync : Queue not found. Delaying for 50000 milliseconds. I, 4076, 1, 2016/09/22, 14:48:08.685, 158348745970, DTAExecutionHost.exe, Main.TryReadAndCacheVssCredentials Acquiring the Token Info I, 4076, 1, 2016/09/22, 14:48:08.841, 158350330678, DTAExecutionHost.exe, Communicating to the Token Servicehttps://.com/tfs/DefaultCollection W, 4076, 1, 2016/09/22, 14:48:10.060, 158362406863, DTAExecutionHost.exe, Service task failed. Retry attempt 10. W, 4076, 1, 2016/09/22, 14:48:13.106, 158392951673, DTAExecutionHost.exe, Service task failed. Retry attempt 9. W, 4076, 1, 2016/09/22, 14:48:16.121, 158423022807, DTAExecutionHost.exe, Service task failed. Retry attempt 8.

I have the Interactive option enabled in the Test Agent Deployment step as seen here: Test Agent Deployment Configuration Step

Any feedback is welcome!

Answer 1

I struggled a lot with running Selenium tests in the Run Functional Tests task, but eventually got it working with a lot of trial-and-error.

First, if I were you, I would try installing an interactive agent onto that machine you're wanting to test with and make sure that it runs locally on that machine with the VS Test task. If it does, then you can narrow the scope of the issue down.

The Selenium tests that I used needed to also point to the DLLs for it to run (I believe they were NUnit in our case). So per the task, I created a run settings file that only specified the testadapterspath tag (make sure to include the dlls into the drop if in RM). When pointing to the runsettings file, be sure to point to the location on the build/release agent, not the test agent location (that tricked me for a bit).

Then, if you are running into issues with DTAExecutionHost crashing (try to double-click on the exe that's installed at C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE) like I had, I think you still need to contact Microsoft from this GitHub post for them to send you a patched exe.

Update: I also notice that the error is pointing to "https://.com/..." and maybe it's not finding your TFS instance properly. I would try running the tests locally via an agent pool and the VS Test task then to see if that still gives the same error.

This may not solve your issue directly but may get you in the right direction. Hope it helps!